Classification of "Near-patient" and "Point-of-Care" SARS-CoV-2 Nucleic Acid Amplification Test Systems and a first approach to evaluate their analytical independence of operator activities.

BACKGROUND: European legislation defines as "near-patient testing" (NPT) what is popularly and in other legislations specified as "point-of-care testing" (POCT). Systems intended for NPT/POCT use must be characterized by independence from operator activities during the analytic procedure. However, tools for evaluating this are lacking. We hypothesized that the variability of measurement results obtained from identical samples with a larger number of identical devices by different operators, expressed as the method-specific reproducibility of measurement results reported in External Quality Assessment (EQA) schemes, is an indicator for this characteristic. MATERIALS AND METHODS: Legal frameworks in the EU, the USA and Australia were evaluated about their requirements for NPT/POCT. EQA reproducibility of seven SARS-CoV-2-NAAT systems, all but one designated as "POCT", was calculated from variabilities in Ct values obtained from the respective device types in three different EQA schemes for virus genome detection. RESULTS: A matrix for characterizing test systems based on their technical complexity and the required operator competence was derived from requirements of the European In Vitro Diagnostic Regulation (IVDR) 2017/746. Good EQA reproducibility of the measurement results of the test systems investigated implies that different users in different locations have no recognizable influence on their measurement results. CONCLUSION: The fundamental suitability of test systems for NPT/POCT use according to IVDR can be easily verified using the evaluation matrix presented. EQA reproducibility is a specific characteristic indicating independence from operator activities of NPT/POCT assays. EQA reproducibility of other systems than those investigated here remains to be determined.

Design of external quality assessment schemes and definition of the roles of their providers in future epidemics.

During an epidemic, individual test results form the basis of epidemiological indicators such as case numbers or incidence. Therefore, the accuracy of measures derived from these indicators depends on the reliability of individual results. In the COVID-19 pandemic, monitoring and evaluating the performance of the unprecedented number of testing facilities in operation, and novel testing systems in use, was urgently needed. External quality assessment (EQA) schemes are unique sources of data reporting on testing performance, and their providers are recognised contacts and support for test facilities (for technical-analytical topics) and health authorities (for planning the monitoring of infection diagnostics). To identify information provided by SARS-CoV-2 genome detection EQA schemes that is relevant for public health microbiology, we reviewed the current literature published in PubMed between January, 2020, and July, 2022. We derived recommendations for EQA providers and their schemes for best practices to monitor pathogen-detection performance in future epidemics. We also showed laboratories, test facilities, and health authorities the information and benefits they can derive from EQA data, and from the non-EQA services of their providers.

Three rounds of a national external quality assessment reveal a link between disharmonic anti-SARS-CoV-2 antibody quantifications and the infection stage.

OBJECTIVES: The WHO's standardized measuring unit, "binding antibody units per milliliter (BAU/mL)," should allow the harmonization of quantitative results by different commercial Anti-SARS-CoV-2 immunoassays. However, multiple studies demonstrate inter-assay discrepancies. The antigenic changes of the Omicron variant affect the performance of Spike-specific immunoassays. This study evaluated the variation of quantitative Anti-SARS-CoV-2-Spike antibody measurements among 46, 50, and 44 laboratories in three rounds of a national external quality assessment (EQA) prior to and after the emergence of the Omicron variant in a diagnostic near-to-real-life setting. METHODS: We analyzed results reported by the EQA participant laboratories from single and sequential samples from SARS-CoV-2 convalescent, acutely infected, and vaccinated individuals, including samples obtained after primary and breakthrough infections with the Omicron variant. RESULTS: The three immunoassays most commonly used by the participants displayed a low intra-assay and inter-laboratory variation with excellent reproducibility using identical samples sent to the participants in duplicates. In contrast, the inter-assay variation was very high with all samples. Notably, the ratios of BAU/mL levels quantified by different immunoassays were not equal among all samples but differed between vaccination, past, and acute infection, including primary infection with the Omicron variant. The antibody kinetics measured in vaccinated individuals strongly depended on the applied immunoassay. CONCLUSIONS: Measured BAU/mL levels are only inter-changeable among different laboratories when the same assay was used for their assessment. Highly variable ratios of BAU/mL quantifications among different immunoassays and infection stages argue against the usage of universal inter-assay conversion factors.

Converting to an international unit system improves harmonization of results for SARS-CoV-2 quantification: Results from multiple external quality assessments.

BACKGROUND: The detection of SARS-CoV-2 vRNA in clinical samples has relied almost exclusively on RT-qPCR as the gold standard test. Published results from various external quality assessments ("ring trials") worldwide have shown that there is still a large variability in results reported for the same samples. As reference standards of SARS-CoV-2 RNA are available, we tested whether using standard curves to convert Ct values into copies/mL (cp/mL) improved harmonization. METHODS: Nine laboratories using 23 test systems (15 of which were unique) prepared standard dilution curves to convert Ct values of 13 SARS-CoV-2 positive samples to cp/mL (hereafter IU/mL). The samples were provided in three rounds of a virus genome detection external quality assessment (EQA) scheme. We tested the precision and accuracy of results reported in IU/mL, and attempted to identify the sources of variability. RESULTS: Reporting results as IU/mL improved the precision of the estimated concentrations of all samples compared to reporting Ct values, although some inaccuracy remained. Variance analysis showed that nearly all variability in data was explained by individual test systems within individual laboratories. When controlling for this effect, there was no significant difference between all other factors tested (test systems, EQA rounds, sample material). CONCLUSIONS: Converting results to copies/mL improved precision across laboratory test systems. However, it seems the results are still very specific to test systems within laboratories. Further efforts could be made to improve accuracy and achieve full harmonization across diagnostic laboratories.

Results of a SARS-CoV-2 virus genome detection external quality assessment round focusing on sensitivity of assays and pooling of samples.

OBJECTIVES: Results of earlier external quality assessment (EQA) rounds suggested remarkable differences in the sensitivity of SARS-CoV PCR assays. Although the test systems are intended to detect SARS-CoV-2 in individual samples, screening is often applied to sample pools to increase efficiency and decrease costs. However, it is unknown to what extent these tests actually meet the manufacturer's specifications for sensitivity and how they perform when testing sample pools. METHODS: The sensitivity of assays in routine use was evaluated with a panel of positive samples in a round of a SARS-CoV-2 virus genome detection EQA scheme. The panel consisted of samples at or near the lower limit of detection ("weakly positive"). Laboratories that routinely test sample pools were asked to also analyze the pooled EQA samples according to their usual pool size and dilution method. RESULTS: All participants could detect a highly positive patient-derived sample (>106 copies/mL). Most (96%) of the test systems could detect at least 1,000 copies/mL, meeting the minimum acceptable benchmark, and many (94%) detected the vRNA in a sample with lower concentration (500 copies/mL). The false negative ratio increased to 16 and 26% for samples with 100 and 50 copies/mL, respectively. CONCLUSIONS: The performance of most assays met or exceeded their specification on sensitivity. If assays are to be used to analyze sample pools, the sensitivity of the assay and the number of pooled samples must be balanced.

Benchmarking medical laboratory performance: survey validation and results for Europe, Middle East, and Africa.

OBJECTIVES: Medical laboratory performance is a relative concept, as are quality and safety in medicine. Therefore, repetitive benchmarking appears to be essential for sustainable improvement in health care. The general idea in this approach is to establish a reference level, upon which improvement may be strived for and quantified. While the laboratory community traditionally is highly aware of the need for laboratory performance and public scrutiny is more intense than ever due to the SARS-CoV-2 pandemic, few initiatives span the globe. The aim of this study was to establish a good practice approach towards benchmarking on a high abstraction level for three key dimensions of medical laboratory performance, generate a tentative snapshot of the current state of the art in the region of Europe, Middle East, and Africa (EMEA), and thus set the stage for global follow-up studies. METHODS: The questionnaire used and previously published in this initiative consisted of 50 items, roughly half relating to laboratory operations in general with the other half addressing more specific topics. An international sample of laboratories from EMEA was approached to elicit high fidelity responses with the help of trained professionals. Individual item results were analyzed using standard descriptive statistics. Dimensional reduction of specific items was performed using exploratory factor analysis and assessed with confirmatory factor analysis, resulting in individual laboratory scores for the three subscales of "Operational performance", "Integrated clinical care performance", and "Financial sustainability". RESULTS: Altogether, 773 laboratories participated in the survey, of which 484 were government hospital laboratories, 129 private hospital laboratories, 146 commercial laboratories, and 14 were other types of laboratories (e.g. research laboratories). Respondents indicated the need for digitalization (e.g. use of IT for order management, auto-validation), automation (e.g. pre-analytics, automated sample transportation), and establishment of formal quality management systems (e.g. ISO 15189, ISO 9001) as well as sustainably embedding them in the fabric of laboratory operations. Considerable room for growth also exists for services provided to physicians, such as "Diagnostic pathways guidance", "Proactive consultation on complex cases", and "Real time decision support" which were provided by less than two thirds of laboratories. Concordantly, the most important kind of turn-around time (TAT) for clinicians, sample-to-result TAT, was monitored by only 40% of respondents. CONCLUSIONS: Altogether, the need for stronger integration of laboratories into the clinical care process became apparent and should be a main trajectory of future laboratory management. Factor analysis confirmed the theoretical constructs of the questionnaire design phase, resulting in a reasonably valid tool for further benchmarking activities on the three aimed-for key dimensions.

Inadequate design of mutation detection panels prevents interpretation of variants of concern: results of an external quality assessment for SARS-CoV-2 variant detection.

OBJECTIVES: Mutation-specific PCR assays have quickly found their way into laboratory diagnostics due to their capacity to be a fast, easy to implement and high-throughput method for the detection of known SARS-CoV-2 variants of concern (VoCs). However, little is known about the performance of such assays in routine laboratory analysis. METHODS: The results reported in a recent round of an external quality assessment (EQA) scheme for SARS-CoV-2 mutation-specific PCR were retrospectively analyzed. For the determination of individual variant-specific sequences as well as for the interpretation results for certain virus variants, correct, incorrect, and unreported results were evaluated, and their possible causes were investigated. RESULTS: A total of 34 laboratories participated in this study. For five samples containing the VoC Alpha + E484K, Beta, Gamma, Delta, or B.1.1.318 (as a variant of interest), 848 results for SARS-2-CoV mutation detection were reported, 824 (97.2%, range per sample 88-100%) of which were correct. Melting curve assays gave 99% correct results, real-time RT-qPCR 94%, microarray-based assays 100%, and MALDI-TOF MS 96%. A total of 122/167 (73%) reported results for SARS-CoV-2 variant determination were correct. Of the 45 inconclusive or incorrect results, 33 (73%) were due to inadequate selection of targets that did not allow identification of contemporary VoC, 11 (24%) were due to incorrect results, and one (3%) was due to correct results of mutation-specific PCR. CONCLUSIONS: Careful and up-to-date selection of the targets used in mutation-specific PCR is essential for successful detection of current SARS-CoV-2 variants.

Benchmarking diagnostic laboratory performance: Survey results for Germany, Austria, and Switzerland.

BACKGROUND AND AIMS: The need for patient safety through consistent diagnostic performance has increasingly been brought into focus during the last two decades. Around the globe operational efficiency of diagnostic laboratories plays a key role in satisfying this need, which has impressively been shown during the recent months of the SARS-CoV­2 pandemic. On a global level, however, there has been a lack to collate and benchmark data for diagnostic laboratories. The goals of this study were to design and pilot a questionnaire addressing key aspects of diagnostic laboratory management. METHODS: The questionnaire was designed using an iterative process and taking into consideration information that could be extracted from the literature, author experience and feedback from informal focus groups of laboratory professionals. The resulting tool consisted of 50 items, either relating to general information or more specifically addressing the topics of "operational performance", "integrated clinical care performance", and "financial sustainability". A limited number of laboratories were surveyed to be able to further improve the newly developed tool and motivate the global laboratory community to participate in further benchmarking activity. RESULTS AND CONCLUSION: Altogether, 65 laboratories participated in the survey, 42 were hospital laboratories and 23 were commercial laboratories. Potential for further improvement and standardization became apparent across the board, e.g. use of IT for order management, auto-validation, or turn-around time (TAT) monitoring. Notably, a gap was identified regarding services provided to physicians, in particular "reflexive test suggestions", "proactive consultation on complex cases", and "diagnostic pathways guidance", which were only provided by about two thirds of laboratories. Concordantly, within-laboratory TAT (Lab TAT) was monitored by about 80% of respondents, while sample-to-result TAT, which is arguably the TAT most relevant to clinicians, was only monitored by 32% of respondents. Altogether, the need for stronger integration of the laboratory into the clinical care process became apparent and should be a main trajectory of future laboratory management.

RT-PCR based SARS-CoV-2 variant screening assays require careful quality control.

BACKGROUND: Distinctive genotypes of SARS-CoV-2 have emerged that are or may be associated with increased transmission, pathogenicity, and/or antibody escape. In many countries, clinical and diagnostic laboratories are under mandate to identify and report these so-called variants of concern (VOC). OBJECTIVES: We used an external quality assessment scheme to determine the scope, accuracy, and reliability of laboratories using various molecular diagnostic assays to identify current VOC (03 March 2021). STUDY DESIGN: Participant laboratories were sent the same five patient-derived samples and were asked to provide their variant detection methods, variant detection results and interpretation of results. RESULTS: Twenty-five laboratories reported a range of RT-qPCR-based assays to identify specific variations in the SARS-CoV-2 spike protein that are characteristic of three VOC lineages. Laboratories that detected VOC-associated nucleotide mutations at four specific sites had the highest ratio of correct classification. Low template copy number and additional variation in target regions resulted in loss of confidence and accuracy in sample classification. CONCLUSIONS: Melting-curve-based assays to identify genomic variants are less time-consuming and require less bioinformatic analysis compared to partial or whole genome sequencing. However, our results suggest that correct classification of a given genotype/lineage (e.g., a VOC) relies on the ability to detect more than one variant site, adequate template in the sample (i.e., relatively high viral load/copy number) and results may be unclear in certain samples with additional genetic variations. These initial results suggest that some diagnostic laboratories may require additional training to interpret and report complex genetic information about a dynamic emerging virus.

The versatility of external quality assessment for the surveillance of laboratory and in vitro diagnostic performance: SARS-CoV-2 viral genome detection in Austria.

OBJECTIVES: External quality assessment (EQA) schemes provide information on individual and general analytical performance of participating laboratories and test systems. The aim of this study was to investigate the use and performance of SARS-CoV-2 virus genome detection systems in Austrian laboratories and their preparedness to face challenges associated with the pandemic. METHODS: Seven samples were selected to evaluate performance and estimate variability of reported results. Notably, a dilution series was included in the panel as a measure of reproducibility and sensitivity. Several performance criteria were evaluated for individual participants as well as in the cohort of all participants. RESULTS: A total of 109 laboratories participated and used 134 platforms, including 67 different combinations of extraction and PCR platforms and corresponding reagents. There were no false positives and 10 (1.2%) false negative results, including nine in the weakly positive sample (Ct ∼35.9, ∼640 copies/mL). Twenty (22%) laboratories reported results of mutation detection. Twenty-five (19%) test systems included amplification of human RNA as evidence of proper sampling. The overall linearity of Ct values from individual test systems for the dilution series was good, but inter-assay variability was high. Both operator-related and systematic failures appear to have caused incorrect results. CONCLUSIONS: Beyond providing certification for participating laboratories, EQA provides the opportunity for participants to evaluate their performance against others so that they may improve operating procedures and test systems. Well-selected EQA samples offer additional inferences to be made about assay sensitivity and reproducibility, which have practical applications.

Variability of cycle threshold values in an external quality assessment scheme for detection of the SARS-CoV-2 virus genome by RT-PCR.

OBJECTIVES: The qualitative results of SARS-CoV-2 specific real-time reverse transcription (RT) PCR are used for initial diagnosis and follow-up of Covid-19 patients and asymptomatic virus carriers. However, clinical decision-making and health management policies often are based additionally on cycle threshold (Ct) values (i.e., quantitative results) to guide patient care, segregation and discharge management of individuals testing positive. Therefore, an analysis of inter-protocol variability is needed to assess the comparability of the quantitative results. METHODS: Ct values reported in a SARS-CoV-2 virus genome detection external quality assessment challenge were analyzed. Three positive and two negative samples were distributed to participating test laboratories. Qualitative results (positive/negative) and quantitative results (Ct values) were assessed. RESULTS: A total of 66 laboratories participated, contributing results from 101 distinct test systems and reporting Ct values for a total of 92 different protocols. In all three positive samples, the means of the Ct values for the E-, N-, S-, RdRp-, and ORF1ab-genes varied by less than two cycles. However, 7.7% of reported results deviated by more than ±4.0 (maximum 18.0) cycles from the respective individual means. These larger deviations appear to be systematic errors. CONCLUSIONS: In an attempt to use PCR diagnostics beyond the identification of infected individuals, laboratories are frequently requested to report Ct values along with a qualitative result. This study highlights the limitations of interpreting Ct values from the various SARS-CoV genome detection protocols and suggests that standardization is necessary in the reporting of Ct values with respect to the target gene.